Causal Back Propagation through Time for Locally Recurrent Neural Networks

This paper concerns dynamic neural networks for signal processing: architectural issues are considered but the paper focuses on learning algorithms that work on-line. Locally recurrent neural networks, namely MLP with IIR synapses and generalization of Local Feedback MultiLayered Networks (LF MLN), are compared to more traditional neural networks, i.e. static MLP with input and/or output buffer (TDNN), FIR MLP and fully recurrent neural networks: simulations results are provided to compare locally recurrent neural networks with TDNN and FIR MLP. Moreover, we propose a new learning algorithm, based on the Back Propagation Through Time and called Causal Back Propagation Through Time (CBPTT), that is faster and more stable than the algorithm previously used for IIR MLP. The algorithm that we propose includes as particular cases the following algorithms: Wan's Temporal Back Propagation, Back Propagation for Sequences (BPS) and Back-Tsoi algorithm.